Various online services utilize computer applications to perform automatic speech recognition (“ASR”) in completing various voice-activated functions initiated from a user's computer, such as the processing of information queries. During speech recognition, training and testing processes are utilized for utterances in order to improve accuracy. However, when training and testing processes are mismatched, speech recognition performance degrades drastically. Current solutions for addressing this problem include the use of speaker adaptation techniques for increasing performance. The speaker adaptation techniques may include the transformation of observed speech features or, alternatively, the adaptation of Hidden Markov Model (“HMM”) speech parameters. Current speaker adaptation techniques, however, have been shown to be deficient with the short utterances often contained in speech utilized for making online voice information queries. It is with respect to these considerations and others that the various embodiments of the present invention have been made.